Surveillance information generation apparatus, imaging direction estimation apparatus, surveillance information generation method, imaging direction estimation method, and program

ABSTRACT

A surveillance information generation apparatus (2000) includes a first surveillance image acquisition unit (2020), a second surveillance image acquisition unit (2040), and a generation unit (2060). The first surveillance image acquisition unit (2020) acquires a first surveillance image (12) generated by a fixed camera (10). The second surveillance image acquisition unit (2040) acquires a second surveillance image (22) generated by a moving camera (20). The generation unit (2060) generates surveillance information (30) relating to object surveillance, using the first surveillance image (12) and first surveillance information (14).

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/406,730, filed May 8, 2019, which is a continuation of U.S.application Ser. No. 15/754,613, filed Feb. 23, 2018, now U.S. Pat. No.10,748,010, issued on Aug. 18, 2020, which is a National Stage ofInternational Application No. PCT/JP2016/063720 filed May 9, 2016,claiming priority based on Japanese Patent Application No. 2015-172082filed Sep. 1, 2015, the disclosures of which are incorporated byreference herein in their entireties.

TECHNICAL FIELD

The present invention relates to a video surveillance.

BACKGROUND ART

In order to surveil or analyze the state of the crowd, a video imaged bya surveillance camera fixed to a building or the like on the movingroute of the crowd is used. For example, Patent Document 1 discloses atechnique for analyzing an input surveillance video to calculate themoving direction of the crowd, and controlling a surveillance deviceaccording to the calculated moving direction.

RELATED DOCUMENT Patent Document

-   [Patent Document 1] Pamphlet of International Publication No.    2014/174737

SUMMARY OF THE INVENTION Technical Problem

In the video imaged by the surveillance camera fixed to the building orthe like as described above, it may be difficult to accurately surveilthe state of the crowd. For example, the surveillance camera fixed tothe building in this way often images the crowd from a distance. In sucha case, since the size of the person captured in the imaged video issmall, it is difficult to recognize the state of the crowd (for example,the number of people in the crowd and its distribution).

The present invention has been made in view of the above problem. Anobject of the present invention is to provide a technique of recognizingthe state of a crowd from an imaged video of a crowd.

Solution to Problem

A surveillance information generation apparatus of the present inventionincludes 1) a first acquisition unit acquiring a first surveillanceimage imaged by a fixed camera, which is a camera a position of which isfixed; 2) a second acquisition unit acquiring a second surveillanceimage imaged by a moving camera, which is a camera a position of whichis not fixed; and 3) a generation unit generating surveillanceinformation of an object by using the first surveillance image and thesecond surveillance image.

An imaging direction estimation apparatus of the present inventionincludes 1) a first moving direction estimation unit estimating a firstmoving direction which is a moving direction of an object in a firstsurveillance image imaged by a fixed camera, which is a camera aposition of which is fixed; 2) a second moving direction estimation unitestimating a second moving direction which is a moving direction of anobject in a second surveillance image imaged by a moving camera, whichis a camera a position of which is not fixed; and 3) an imagingdirection estimation unit estimating the imaging direction of the movingcamera, based on the first moving direction, the second movingdirection, the position and pose of the fixed camera, and the positionof the moving camera.

A first surveillance information generation method of the presentinvention is executed by a computer. The method includes 1) a firstacquisition step of acquiring a first surveillance image imaged by afixed camera, which is a camera a position of which is fixed; 2) asecond acquisition step of acquiring a second surveillance image imagedby a moving camera, which is a camera a position of which is not fixed;and 3) a generation step of generating surveillance information of anobject by using the first surveillance image and the second surveillanceimage.

An imaging direction estimation method of the present invention isexecuted by a computer. The method includes 1) a first moving directionestimation step of estimating a first moving direction which is a movingdirection of an object in a first surveillance image imaged by a fixedcamera, which is a camera a position of which is fixed; 2) a secondmoving direction estimation step of estimating a second moving directionwhich is a moving direction of an object in a second surveillance imageimaged by a moving camera, which is a camera a position is not fixed;and 3) an imaging direction estimation step of estimating the imagingdirection of the moving camera, based on the first moving direction, thesecond moving direction, the position and pose of the fixed camera, andthe position of the moving camera.

Advantageous Effects of Invention

According to the present invention, a technique of recognizing the stateof a crowd from an imaged video of the crowd is provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages will becomemore apparent from the following description of preferred exemplaryembodiments and the accompanying drawings.

FIG. 1 is a block diagram illustrating a surveillance informationgeneration apparatus according to Example Embodiment 1.

FIG. 2 is a diagram conceptually illustrating an operation of thesurveillance information generation apparatus of Example Embodiment 1.

FIG. 3 is a flowchart illustrating a flow of a process executed by thesurveillance information generation apparatus of Example Embodiment 1.

FIG. 4 is a diagram illustrating a hardware configuration of a computerthat implements the surveillance information generation apparatus ofExample Embodiment 1.

FIG. 5 is a diagram illustrating a display in which a secondsurveillance image is superimposed on a first surveillance image.

FIG. 6 illustrates a scene in which second surveillance images aredisplayed side by side in the vicinity of a left end of a displayscreen.

FIG. 7 illustrates a scene in which the first surveillance image and thesecond surveillance image are displayed side by side on the displayscreen.

FIG. 8 is a diagram illustrating a scene in which a mark indicating amoving camera is displayed on the first surveillance image.

FIG. 9 is a diagram illustrating a display in a case where a mousecursor is clicked in FIG. 8 .

FIG. 10 is a diagram illustrating a scene in which an imaging directionof the moving camera is displayed.

FIG. 11 is a first diagram illustrating superposition of a display basedon distribution information on the first surveillance image.

FIG. 12 is a second diagram illustrating superposition of a displaybased on distribution information on the first surveillance image.

FIGS. 13A and 13B are diagrams illustrating overlap between a rangecaptured in the first surveillance image and a range captured in thesecond surveillance image.

FIG. 14 is a block diagram illustrating a surveillance informationgeneration apparatus including a map information acquisition unit.

FIG. 15 is a diagram illustrating a map displayed on a display screen.

FIG. 16 is a diagram illustrating a map on which a heat map generatedbased on the corrected distribution information is superimposed.

FIG. 17 is a block diagram illustrating a surveillance informationgeneration apparatus according to Example Embodiment 2.

FIG. 18 is a diagram conceptually illustrating an operation of thesurveillance information generation apparatus of Example Embodiment 2.

FIG. 19 is a flowchart illustrating a flow of a process executed by thesurveillance information generation apparatus of Example Embodiment 2.

FIG. 20 is a diagram illustrating an optical flow calculated for a firstsurveillance image.

FIG. 21 is a diagram illustrating a change in the position of an object.

FIG. 22 is a diagram for explaining an operation of an imaging directionestimation unit.

FIG. 23 is a diagram for explaining a method of determining a candidateimaging direction with reference to a direction of a sidewalk.

FIG. 24 is a diagram illustrating a case where the imaging ranges of thefixed camera and the moving camera do not overlap.

FIG. 25 is a diagram for explaining a method of estimating a movingdirection on a plane of a crowd using map information.

FIG. 26 is a diagram illustrating a case where the crowd has a pluralityof moving routes.

FIG. 27 is a block diagram illustrating a surveillance informationgeneration apparatus including a moving route information acquisitionunit.

FIG. 28 is a diagram for explaining a method of estimating a movingdirection of the crowd using moving route information.

FIG. 29 is a diagram for explaining a method of narrowing down candidateimaging directions using an electronic compass.

FIG. 30 is a diagram for explaining a method of narrowing down candidateimaging directions based on the background captured in the secondsurveillance image.

FIGS. 31A and 31B are diagrams for explaining a method of narrowing downcandidate imaging directions based on the position of a specificbackground on the second surveillance image.

FIGS. 32A and 32B are diagrams illustrating an example in which the flowof the crowd changes near an intersection.

FIG. 33 is a diagram illustrating breakdown of an estimation process ofthe imaging direction of the moving camera executed by the imagingdirection estimation unit in time series.

FIG. 34 is a diagram illustrating a change in the position of featurepoints of the second surveillance image.

FIG. 35 is a block diagram illustrating an imaging direction estimationapparatus.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings. In all the drawings, the same components aredenoted by the same reference numerals, and the description thereof willnot appropriately be repeated.

Example Embodiment 1

FIG. 1 is a block diagram illustrating a surveillance informationgeneration apparatus 2000 according to Example Embodiment 1. Further, inFIG. 1 , each block represents a functional unit configuration, insteadof a hardware unit configuration.

The surveillance information generation apparatus 2000 uses two types ofsurveillance images such as a surveillance image generated by a fixedcamera and a surveillance image generated by a moving camera. The fixedcamera is a camera the position of which is fixed. For example, thefixed camera is a surveillance camera which is fixedly installed invarious places such as walls, pillars, or ceilings. Note that, the placewhere the fixed camera is installed may be indoors or outdoors. Inaddition, with respect to the wall or the like on which the fixed camerais provided, it is not limited to real estate as long as the positionthereof is fixed for a certain period of time. For example, the wall onwhich the fixed camera is installed may be partitions, pillars, or thelike which are temporarily installed at the event venue or the like.

The moving camera is a camera the position of which is moved. Forexample, the moving camera is worn on a person, or is attached to a car,a motorcycle, or a flying object, or the like. The moving camera is wornon a person is, for example, a camera held by hand (a camera of a mobileterminal such as a video camera or a smartphone), or a camera fixed to ahead or a chest, or the like (such as a body-worn camera). The cameraattached to a car, a motorcycle, or a flying object, or the like may bea camera attached for use as a so-called drive recorder, or a cameraattached separately for surveillance imaging.

Both the moving camera and the fixed camera capture a place to besurveilled as a video. The place to be surveilled is arbitrary. Forexample, the place to be surveilled is the route between the event venueand the nearest station. Note that, the place to be surveilled may beindoors or outdoors. The imaging range of the moving camera and theimaging range of the fixed camera may or may not overlap each other.

FIG. 2 is a diagram conceptually illustrating an operation of thesurveillance information generation apparatus 2000. The fixed camera 10images the crowd and generates a first surveillance image 12. The crowdhere means one or more objects. The object may be a person or a thingother than a person (for example, a car, a motorbike, an animal, or thelike). The moving camera 20 images the crowd and generates a secondsurveillance image 22. Note that, the crowd imaged by the fixed camera10 and the crowd imaged by the moving camera 20 may be the same ordifferent.

The surveillance information generation apparatus 2000 generatessurveillance information 30 using the first surveillance image 12 andthe second surveillance image 22. The surveillance information 30 isinformation on object surveillance. The details contents of thesurveillance information 30 and a generation method thereof will bedescribed later.

In order to implement the above operation, the surveillance informationgeneration apparatus 2000 includes a first surveillance imageacquisition unit 2020, a second surveillance image acquisition unit2040, and a generation unit 2060. The first surveillance imageacquisition unit 2020 acquires first surveillance image 12. The secondsurveillance image acquisition unit 2040 acquires second surveillanceimage 22. The generation unit 2060 generates surveillance information30, using the first surveillance image 12 and the first surveillanceinformation 14.

Advantageous Effect

According to the present example embodiment, surveillance information ofcrowd is generated using the surveillance image generated by the movingcamera 20 in addition to the surveillance image generated by the fixedcamera 10. Thus, compared with the case where only the fixed camera 10is used to recognize the state of the crowd, it becomes possible torecognize the state of the crowd more accurately.

Hereinafter, the present example embodiment will be described in moredetail.

<Flow of Process>

FIG. 3 is a flowchart illustrating the flow of a process executed by thesurveillance information generation apparatus 2000 of ExampleEmbodiment 1. The first surveillance image acquisition unit 2020acquires a first surveillance image 12 (S102). The second surveillanceimage acquisition unit 2040 acquires a second surveillance image 22(S104). The generation unit 2060 generates surveillance information 30,using the first surveillance image 12 and the second surveillance image22 (S106).

<Hardware Configuration of Surveillance Information Generation Apparatus2000>

FIG. 4 is a diagram illustrating a hardware configuration of a computer1000 that implements the surveillance information generation apparatus2000 of Example Embodiment 1. The computer 1000 may be implemented usinga special-purpose apparatus dedicated for implementing the surveillanceinformation generation apparatus 2000, or may be implemented using ageneral-purpose apparatus such as a personal computer (PC) or a portableterminal.

The computer 1000 includes a bus 1020, processor 1040, a memory 1060, astorage 1080, an input and output interface 1100, and a networkinterface 1120. The bus 1020 is a data transmission path through whichthe processor 1040, the memory 1060, the storage 1080, the input andoutput interface 1100, and the network interface 1120 mutually transmitand receive data. However, a method of connecting the processors 1040and the like to each other is not limited to bus connection. Theprocessor 1040 is a processor such as a central processing unit (CPU) ora graphics processing unit (GPU). The memory 1060 is a memory such as arandom access memory (RAM) or a read only memory (ROM). The storage 1080is a storage apparatus such as a hard disk, a solid state drive (SSD),or a memory card. Further, the storage 1080 may be a memory such as aRAM or a ROM.

The input and output interface 1100 is an interface for connecting thecomputer 1000 and an input/output device. For example, a keyboard, amouse, or the like is connected to the input and output interface 1100.

The network interface 1120 is an interface for communicably connectingthe computer 1000 to an external apparatus. The network interface 1120may be a network interface for connection with a wired line, or anetwork interface for connection with a wireless line. For example, thecomputer 1000 that implements the surveillance information generationapparatus 2000 is connected to the fixed camera 10 or the moving camera20 through a network. However, a method of connecting the computer 1000to the fixed camera 10 or the moving camera 20 is not limited to theconnection through the network. The computer 1000 may not becommunicably connected to the fixed camera 10 or the moving camera 20.

The storage 1080 stores a program module for implementing each of thefunctions of the surveillance information generation apparatus 2000. Byexecuting these respective program modules, the processor 1040implements each of the functions corresponding to the program modules.Here, when executing each of the above modules, the processor 1040 mayexecute the modules after reading them on the memory 1060, or mayexecute the modules without reading them on the memory 1060.

The hardware configuration of the computer 1000 is not limited to theconfiguration illustrated in FIG. 4 . For example, each program modulemay be stored in the memory 1060. In this case, the computer 1000 maynot include the storage 1080.

<Details of First Surveillance Image Acquisition Unit 2020>

The first surveillance image acquisition unit 2020 acquires a firstsurveillance image 12 (S102). Here, there are various methods by whichthe first surveillance image acquisition unit 2020 acquires the firstsurveillance image 12. For example, the first surveillance imageacquisition unit 2020 receives the first surveillance image 12transmitted from the fixed camera 10. In another example, the firstsurveillance image acquisition unit 2020 may access the fixed camera 10,and acquire the first surveillance image 12 stored in the fixed camera10. Note that the fixed camera 10 may store the first surveillance image12 in a storage apparatus provided outside the fixed camera 10. In thiscase, the first surveillance image acquisition unit 2020 may acquire thefirst surveillance image 12 by accessing the storage apparatus.

The first surveillance image acquisition unit 2020 may acquire the firstsurveillance image 12 in real time, or may acquire the firstsurveillance image 12 for a while after the generation of the firstsurveillance image 12. In the latter case, for example, the surveillanceinformation generation apparatus 2000 acquires the first surveillanceimage 12 and the second surveillance image 22 taken in the past (forexample, on the previous day) and generates the surveillance informationon the past surveillance image to analyze crowd behaviors and others.

<Details of Second Surveillance Image Acquisition Unit 2040>

The second surveillance image acquisition unit 2040 acquires a secondsurveillance image 22 (S104). Here, the method by which the secondsurveillance image acquisition unit 2040 acquires the secondsurveillance image 22 is the same as the method by which the firstsurveillance image acquisition unit 2020 acquires the first surveillanceimage 12.

<Details of Generation Unit 2060>

The generation unit 2060 generates surveillance information 30 of theobject, using the first surveillance image 12 and the secondsurveillance image 22 (S106). As described above, the object is notlimited to only people, but may be arbitrary. What the generation unit2060 handles as an object may be previously set in the generation unit2060, may be stored in a storage apparatus or the like accessible fromthe generation unit 2060, or may be set manually when the generationunit 2060 operates.

There are a variety of the surveillance information 30 generated by thegeneration unit 2060. Below, a specific example of the surveillanceinformation 30 and each generation method thereof will be described.

Specific Example 1 of Surveillance Information 30

The generation unit 2060 generates a display in which the secondsurveillance image 22 is superimposed on the first surveillance image12, as the surveillance information 30. FIG. 5 is a diagram illustratinga display in which the second surveillance image 22 is superimposed onthe first surveillance image 12. The display screen 40 of FIG. 5displays a screen in which a second surveillance image 22-1 to a secondsurveillance image 22-3 are superimposed on the first surveillance image12.

The display screen 40 is viewed by, for example, a surveillant in asecurity room. By viewing the display screen 40, the surveillant canrecognize the detailed scene of the individual places taken by themoving camera 20 while recognizing the overall scene of the surveillanceplace taken by the fixed camera 10. Therefore, the surveillant andothers can recognize the state of the crowd flexibly and accurately.

The position of the second surveillance image 22 provided on the firstsurveillance image 12 is preferably a position or near the position onthe first surveillance image 12 corresponding to the position of thesecond surveillance image 22 on the real world. In this case, thegeneration unit 2060 determines the position at which the secondsurveillance image 22 is superimposed, using position information of themoving camera 20, the position information of the fixed camera 10, andthe camera parameter representing the pose of the fixed camera 10.Specifically, the generation unit 2060 uses the position information andpose of the fixed camera to determine the position corresponding to theposition information of the moving camera 20 from among the placescaptured in the first surveillance image 12. Here, the pose of the fixedcamera 10 includes the horizontal direction and the vertical directionof the imaging direction of the fixed camera 10.

The position information of each camera is arbitrary information thatcan specify the position of the camera. For example, the positioninformation of the camera is information indicating the globalpositioning system (GPS) coordinates of the camera.

There are various methods by which the generation unit 2060 acquires theposition information of the moving camera 20. The position informationof the moving camera 20 is included in, for example, the metadata of thesecond surveillance image 22. In this case, the generation unit 2060acquires the position information of the moving camera 20 from themetadata of the second surveillance image 22. For example, thegeneration unit 2060 may receive position information separatelytransmitted by the moving camera 20. The transmission may be performedvoluntarily by the moving camera 20, or may be performed in response toa request from the generation unit 2060.

The method by which the generation unit 2060 acquires the positioninformation of the fixed camera 10 is the same as for example, themethod by which the generation unit 2060 acquires the positioninformation of the moving camera 20. Since the position of the fixedcamera 10 is fixed, the position information of the fixed camera 10 maybe stored in advance in the storage unit accessible from the generationunit 2060. Further, for example, the position information of the fixedcamera 10 may be manually input to the generation unit 2060.

Note that, the position of the second surveillance image 22 on the firstsurveillance image 12 is not limited to the position based on theposition of the second surveillance image 22 in the real world. Forexample, the second surveillance image 22 may be displayed at apredetermined position of the first surveillance image 12. FIG. 6illustrates a scene in which second surveillance images are displayedside by side in the vicinity of a left end of the display screen 40. Thepredetermined position may be previously set in the generation unit 2060or may be stored in the storage apparatus accessible from the generationunit 2060.

The display position of the second surveillance image 22 may bechangeable by the user's operation. For example, the surveillanceinformation generation apparatus 2000 receives an operation such asdragging the second surveillance image 22 with a mouse and changes thedisplay position of the second surveillance image 22 in response to theoperation.

Further, the generation unit 2060 may display the first surveillanceimage 12 and the second surveillance image 22 side by side instead ofsuperimposing the second surveillance image 22 on the first surveillanceimage 12. FIG. 7 illustrates a scene in which the first surveillanceimage 12 and the second surveillance image 22 are displayed side by sideon the display screen 40.

In the case where the display position of the second surveillance image22 on the first surveillance image 12 is not the position based on theposition of the moving camera 20 in the real world or the secondsurveillance image 22 is not superimposed on the first surveillanceimage 12, it is preferable to know the position of the moving camera 20which images each second surveillance image 22. In this case, forexample, the generation unit 2060 displays a mark representing eachmoving camera 20, which will be described later, at the position on thefirst surveillance image 12 corresponding to the position of each movingcamera 20 in the real world. Then, the generation unit 2060 displaysinformation (for example, such as the mark number) indicating to whichmark each second surveillance image 22 corresponds, for example, next tothe second surveillance image 22.

Specific Example 2 of Surveillance Information 30

The generation unit 2060 generates a display in which a mark indicatingthe moving camera 20 is superimposed on the first surveillance image 12.Furthermore, in a case where the mark is selected by the user (such as asurveillant), the generation unit 2060 displays the second surveillanceimage 22 generated by the moving camera 20 corresponding to the mark. Asa result, surveillance information 30, which is a display in which thesecond surveillance image 22 is superimposed on the first surveillanceimage 12, is generated as in the case of the aforementioned specificexample 1.

In this manner, since the second surveillance image 22 is displayed inresponse to selection by a surveillant or the like, as compared with thecase where all the second surveillance images 22 are unconditionallydisplayed, it becomes easy to recognize the detailed scene of the crowdat the place where the surveillant or others wants to watch, whilemaking it easy to recognize the overall scene of the crowd captured inthe first surveillance image 12.

FIG. 8 is a diagram illustrating a scene in which a mark indicating themoving camera 20 is displayed on the first surveillance image 12. InFIG. 8 , there are three moving cameras 20-1 to 3 within the imagingrange of the fixed camera 10, and their positions are represented bymarks 50-1 to 3, respectively. The position of each mark 50 in FIG. 8 isa position corresponding to the position of the corresponding movingcamera 20 in the real world.

For example, the user selects the mark 50 using the mouse. In FIG. 8 ,the mouse cursor 60 is located on the mark 50-3. When the user clicksthe mouse cursor 60 in this state, the generation unit 2060 displays thesecond surveillance image 22-3 generated by the moving camera 20-3 onthe display screen 40. FIG. 9 is a diagram illustrating a display in acase where the mouse cursor 60 is clicked in FIG. 8 .

The user's selection operation on the mark 50 is not limited to themouse operation. For example, there is an operation to select the mark50 using a touch panel or a keyboard as another operation.

The display position of the mark of the moving camera 20 displayed onthe first surveillance image 12 is not limited to the positioncorresponding to the position of the moving camera 20 in the real world.This point is the same as the display position of the secondsurveillance image 22 on the first surveillance image 12 describedabove.

It is preferable that the imaging direction of each moving camera 20 isfurther displayed on the display screen 40. By doing so, the surveillantor the like who sees the display screen 40 can more accurately andeasily recognize the place of the scenery represented by the secondsurveillance image 22. FIG. 10 is a diagram illustrating a scene inwhich an imaging direction of the moving camera 20 is displayed. In FIG.10 , the direction indicated by the imaging direction 52 is the imagingdirection of the moving camera 20 corresponding to the mark 50.

Here, there are various ways in which the generation unit 2060recognizes the imaging direction of each moving camera 20. In a casewhere an electronic compass is built in the moving camera 20 or in amobile terminal integrated with the moving camera 20, the generationunit 2060 sets the direction indicated by the output of the electroniccompass as the imaging direction of the moving camera 20. For example,the generation unit 2060 may use the imaging direction of the movingcamera 20 estimated by the method described in an example embodiment tobe described later.

Specific Example 3 of Surveillance Information 30

The generation unit 2060 generates, as the surveillance information 30,a display in which information on the crowd is superimposed on the firstsurveillance image 12. The information on the crowd is, for example,information indicating the distribution of objects included in thecrowd. In the following, the information indicating the distribution ofobjects included in the crowd is represented as distributioninformation.

FIG. 11 and FIG. 12 are diagrams illustrating superposition of a displaybased on the distribution information on the first surveillance image12. On the display screen 40 of FIG. 11 , a heat map 61 representing thedistribution of people is superimposed on the first surveillance image12. For example, the heat map is a heat map in which a place having ahigh degree of congestion of people is red and a place having a lowdegree of congestion of people is blue.

On the display screen 40 of FIG. 12 , the place where the density ofpeople is high is highlighted with a thick frame line 62. Here, in FIG.12 , the first surveillance image 12 is divided into a plurality ofpartial regions. Specifically, the first surveillance image 12 isdivided into 24 partial regions by dividing the first surveillance image12 vertically into six equal parts and horizontally into four equalparts. Note that, a dotted line representing each partial region isdisplayed in FIG. 12 to facilitate understanding of the drawing, butactually the dotted line may not be displayed.

The generation unit 2060 generates, as the surveillance information 30,a display in which the frame line 62 emphasizing an area having a highdegree of congestion of people is superimposed on the first surveillanceimage 12. Specifically, the generation unit 2060 generates distributioninformation indicating the number of people captured in each partialregion, and sets the partial region in which the number is equal to orlarger than a predetermined value as a region having a high degree ofcongestion of people.

By superimposing on the first surveillance image 12, the distributioninformation of the crowd generated by using the first surveillance image12 and the second surveillance image 22 in this manner, the surveillantor the like is able to easily recognize the state of the crowd that isdifficult to recognize, only by seeing the first surveillance image 12and the second surveillance image 22.

<<<How to Generate Distribution Information>>>

It will be described how distribution information is generated forrealizing each of the above-mentioned displays. The generation unit 2060generates distribution information, using the first surveillance image12 and the second surveillance image 22. First, the generation unit 2060generates distribution information on the first surveillance image 12 byperforming a process such as an image recognition process on the firstsurveillance image 12. More specifically, the generation unit 2060divides the first surveillance image 12 into a plurality of partialregions, and calculates the number of objects captured in each partialregion. As a result, distribution information indicating the number ofobjects for each partial region of the first surveillance image 12 isgenerated.

Further, the generation unit 2060 determines the area captured in thesecond surveillance image 22 among the partial regions of the firstsurveillance image 12. The generation unit 2060 corrects the number ofobjects indicated for the determined area in the distributioninformation, by using the number of objects in the second surveillanceimage 22. Then, the generation unit 2060 displays a superposition of thecorrected distribution information on the first surveillance image 12.Note that, the number of objects captured in the second surveillanceimage 22 can be calculated by performing an image recognition process orthe like on the second surveillance image 22 in the same manner as thenumber of objects captured in the first surveillance image 12.

FIGS. 13A and 13B are diagrams illustrating overlap between a rangecaptured in the first surveillance image 12 and a range captured in thesecond surveillance image 22. In FIGS. 13A and 13B, the partial region64-1 is the above-mentioned partial region obtained by dividing thefirst surveillance image 12 into a plurality of parts. In FIG. 13A, therange 65 captured in the second surveillance image 22 is within onepartial region 64-1. In this case, the generation unit 2060 calculatesthe number of objects captured in the entire second surveillance image22, and corrects the number of objects in the partial region 64-1indicated by the distribution information by using the calculated numberof objects. For example, the generation unit 2060 performs a processsuch as 1) replacing the number of objects in the partial region 64-1indicated by the distribution information with the number of objectscalculated for the second surveillance image 22, or 2) replacing thenumber of objects in the partial region 64-1 indicated by thedistribution information with the statistical value of that number andthe number of objects calculated for the second surveillance image 22.The statistical value in 2) is a weighted average with a weight greaterthan a weight for the number of objects indicated by the distributioninformation, with respect to the number of objects calculated for thesecond surveillance image 22.

In the case of FIG. 13B, the range 65 captured in the secondsurveillance image 22 straddles the two partial regions 64-1 and 64-2.In this case, the generation unit 2060 divides the second surveillanceimage 22 into an area A overlapping the partial region 64-1 and an areaB overlapping the partial region 64-2, and calculates the number ofobjects captured in each area. Then, the generation unit 2060 correctsthe number of objects in the partial region 64-1 indicated by thedistribution information by using the number of objects calculated forthe region A. Similarly, the generation unit 2060 corrects the number ofobjects in the partial region 64-2 indicated by the distributioninformation by using the number of objects calculated for the region B.Each correction method is the same as the method described using FIG.13A.

In this way, by correcting the distribution information of the objectcalculated for the first surveillance image 12 by using the number ofobjects captured in the second surveillance image 22, the distributionof the object can be calculated more accurately. This is because thenumber of objects calculated for the second surveillance image 22 can becalculated more accurately than the number of objects calculated for thefirst surveillance image 12.

For example, as shown in FIG. 11 , in a case where the fixed camera 10images a wide range from a distance far from the moving camera 20, sincethe captured object is small in the first surveillance image 12, it maybe difficult to calculate the number of the captured objects accurately.On the other hand, the object is captured largely and clearly in thesecond surveillance image 22 generated by the moving camera 20 imagingthe object at a distance closer than the fixed camera 10. Therefore, thegeneration unit 2060 can calculate the number of objects captured in thesecond surveillance image 22 more accurately than the number of objectscaptured in the first surveillance image 12.

In addition, for example, as shown in FIG. 11 , in a case where thefixed camera 10 performs imaging at such an angle that it looks downobliquely from a distance, people who are close to each other mayoverlap each other and some people may not be captured in the firstsurveillance image 12 in some cases. On the other hand, for example, byattaching the moving camera 20 to a compact flying object and performingimaging the crowd from directly above, it is possible to perform imagingsuch that people in close proximity do not overlap. Therefore, thenumber of objects calculated using the second surveillance image 22generated by the moving camera 20 performing imaging in this way is moreaccurate than the number of objects calculated using the firstsurveillance image 12.

<<<about Update of Distribution Information>>>

Since the fixed camera 10 and the moving camera 20 are cameras thatperform imaging and generate videos, the first surveillance image 12 andthe second surveillance image 22 are repeatedly generated. Thus, thesurveillance information generation apparatus 2000 may repeatedlygenerate the above-mentioned distribution information and update thedistribution information to be displayed. This makes it possible todisplay distribution information such as a heat map like an animation.Note that, distribution information may be generated using all the firstsurveillance images 12 and the second surveillance images 22, or may begenerated using some of the first surveillance images 12 and the secondsurveillance images 22. In the latter case, distribution information isgenerated at intervals such as once per second or once per seconds.Further, the surveillance information generation apparatus 2000 mayreceive a user operation instructing the generation of distributioninformation, and update the distribution information only when receivingthe user operation.

<<<Other Use Methods of Distribution Information>>>

In the above example, the generation unit 2060 handles the display inwhich the distribution information is superimposed on the firstsurveillance image 12 as the surveillance information 30. However, thegeneration unit 2060 may use the distribution information itself as thesurveillance information 30. In this case, for example, the generationunit 2060 stores the distribution information in the storage apparatusor the like, or displays the distribution information on the displayscreen in a tabular form, a graph form or the like. The distributioninformation can be used for behavior analysis of the crowd, or the like.Further, the surveillance information generation apparatus 2000 may usedistribution information as another specific example described below.

Specific Example 4 of Surveillance Information 30

The generation unit 2060 may superimpose the aforementioned distributioninformation on the map of the place to be surveilled and display it. Inthis case, the surveillance information generation apparatus 2000includes a map information acquisition unit 2080. The map informationacquisition unit 2080 acquires map information, which is informationrelating to a map around the place to be surveilled. FIG. 14 is a blockdiagram illustrating the surveillance information generation apparatus2000 including the map information acquisition unit 2080. For example,the map information indicates the positions of a sidewalk, a road, abuilding, and the like. For example, it is assumed that the mapinformation is stored in advance in a storage apparatus accessible fromthe surveillance information generation apparatus 2000 or the like.

FIG. 15 is a diagram illustrating a map 200 displayed on the displayscreen 40. The surveillance target in FIG. 15 is an indoor floor. InFIG. 15 , a plurality of fixed cameras 10 are installed. In addition,there are a plurality of moving cameras 20. In FIG. 15 , the position ofthe moving camera 20 is indicated by a mark 50. Note that, the positionof the fixed camera 10 and the position of the moving camera 20 on themap 200 can be calculated using the position information of the fixedcamera 10, the position information of the moving camera 20, and theposition information of the place indicated by the map 200. Further, ina case where the arrangement of the fixed camera 10 is fixed, theposition of the fixed camera 10 may be indicated in the map 200 inadvance.

First, the generation unit 2060 uses the first surveillance image 12generated by each fixed camera 10 to generate distribution informationof the object in the imaging range of each fixed camera 10. Furthermore,the generation unit 2060 corrects each piece of distribution informationby using the number of objects calculated for the second surveillanceimage 22. Note that, a method of correcting the distribution informationusing the number of objects calculated for the second surveillance image22 is the same as the method described using FIGS. 13A and 13B.

The generation unit 2060 superimposes a display based on each piece ofcorrected distribution information on the map 200 and displays it. FIG.16 is a diagram illustrating the map 200 on which a heat map generatedbased on the corrected distribution information is superimposed.

If the distribution information is displayed on the map as describedabove, the background is simplified, and thus it becomes easy for asurveillant or the like to visually recognize the distributioninformation. If distribution information is displayed on the map, thestate of the crowd in a surveillance place can be displayed on onescreen by the plurality of fixed cameras 10. Therefore, a surveillant orthe like can easily recognize the scene of a crowd who are distributedover a wide range.

Note that, the generation unit 2060 may receive an operation ofselecting the mark 50 from the user and display the second surveillanceimage 22 generated by the moving camera 20 corresponding to the selectedmark 50 on the map 200, similar to the process described with referenceto FIG. 8 and FIG. 9 . Further, the generation unit 2060 may display onthe map 200, distribution information (information indicating thedistribution of objects in the imaging range of the moving camera 20)generated using the second surveillance image 22, instead of the secondsurveillance image 22 or together with the second surveillance image 22.

Modification Example

The generation unit 2060 may generate the aforementioned distributioninformation using only the second surveillance image 22 without usingthe first surveillance image 12. In this case, for example, thegeneration unit 2060 calculates the number of objects captured in thesecond surveillance image 22 generated by each of the plurality ofmoving cameras 20, and generates distribution information as a list ofthe combination of “the imaging range of the second surveillance image22 and the number of objects”. Then, the generation unit 2060 generatesa heat map or the like using the distribution information, superimposesthe generated heat map on the first surveillance image 12 or the map200, and displays it.

Further, the generation unit 2060 may generate distribution informationfor each second surveillance image 22 in the same method as the methodfor generating distribution information for the first surveillance image12. In this case, the generation unit 2060 generates a heat map or thelike using a plurality of pieces of distribution information,superimposes the generated heat map on the first surveillance image 12or the map 200, and displays it.

Example Embodiment 2

FIG. 17 is a block diagram illustrating a surveillance informationgeneration apparatus 2000 according to Example Embodiment 2. In FIG. 17, each block represents a functional unit configuration, instead of ahardware unit configuration.

The surveillance information generation apparatus 2000 of ExampleEmbodiment 2 has a function of estimating the imaging direction of themoving camera 20. In the case of the fixed camera 10 provided on a wallor the like, generally, the imaging direction of the fixed camera 10 isknown by acquiring camera parameters (such as a rotation angle in ahorizontal direction and a rotation angle in a vertical direction)representing the current pose of the camera. For example, it is assumedthat there is a fixed camera 10 provided facing north in an initialstate, and the camera parameter acquired from the fixed camera 10represents a rotation angle in the horizontal direction of +45 degrees.Here, the rotation angle in the horizontal direction is represented withthe counterclockwise direction as the positive direction. This pointalso applies to the following description. Then, the imaging directionof the fixed camera 10 can be recognized to be northwest, which is adirection rotated counterclockwise by 45 degrees from the north.

On the other hand, in the case of the moving camera 20 worn by a personor the like, unlike the fixed camera 10, it is difficult to determine areference orientation of the initial state, and thus it is difficult todetermine the imaging direction of the moving camera 20.

Here, there is also a method to determine the imaging direction byproviding an electronic compass in a camera or the like. However, withthis method, if the precision of the electronic compass is poor, it isimpossible to determine an accurate imaging direction. Further, if ahigh-precision electronic compass needs to be attached to the movingcamera 20, there is a possibility that the manufacturing cost of themoving camera 20 may increase. Furthermore, if the high-precisionelectronic compass needs to be attached to the moving camera 20, itbecomes difficult to use a general-purpose camera such as a camera of asmartphone or a handy camera as the moving camera 20.

Therefore, the surveillance information generation apparatus 2000 of thepresent example embodiment estimates the imaging direction of the movingcamera 20 using the first surveillance image 12 and the secondsurveillance image 22. FIG. 18 is a diagram conceptually illustrating anoperation of the surveillance information generation apparatus 2000 ofExample Embodiment 2. The surveillance information generation apparatus2000 acquires the first surveillance image 12 generated by the fixedcamera 10, and estimates the moving direction (hereinafter, referred toas the first moving direction) of the crowd in the first surveillanceimage 12. The surveillance information generation apparatus 2000acquires the second surveillance image 22 generated by the moving camera20, and estimates the moving direction (hereinafter, referred to as thesecond moving direction) of the crowd in the second surveillance image22. Then, the surveillance information generation apparatus 2000estimates the imaging direction of the moving camera 20, based on themoving direction of the crowd in the first surveillance image 12 and themoving direction of the crowd in the second surveillance image 22.

In order to implement the above operation, the surveillance informationgeneration apparatus 2000 of Example Embodiment 2 further includes afirst moving direction estimation unit 2100, a second moving directionestimation unit 2120, and an imaging direction estimation unit 2140. Thefirst moving direction estimation unit 2100 estimates the movingdirection of the crowd in the first surveillance image 12. The secondmoving direction estimation unit 2120 estimates the moving direction ofthe crowd in the second surveillance image 22. The imaging directionestimation unit 2140 estimates the imaging direction of the movingcamera 20, based on the first moving direction, the second movingdirection, the position and pose of the fixed camera 10, and theposition of the moving camera 20.

Advantageous Effect

According to the surveillance information generation apparatus 2000 ofthe example embodiment, since the moving direction of the moving camera20 is estimated using the first surveillance image 12 and the secondsurveillance image 22, it is possible to accurately recognize theimaging direction of the moving camera 20 even in a case where theimaging direction of the moving camera 20 cannot be calculatedaccurately using a device such as an electronic compass attached to themoving camera 20.

The imaging direction of the moving camera 20 estimated by thesurveillance information generation apparatus 2000 can be used forprocesses such as a process of visualizing the imaging direction of themoving camera 20 described in Example Embodiment 1, and a process ofmapping the range captured in the second surveillance image 22 on thefirst surveillance image 12 or the map. However, the usage of theestimated imaging direction of the moving camera 20 is arbitrary, and itis not limited to a way of use described in Example Embodiment 1.

Hereinafter, the surveillance information generation apparatus 2000 ofthe present example embodiment will be described in more detail.

<Flow of Process>

FIG. 19 is a flowchart illustrating the flow of a process executed bythe surveillance information generation apparatus 2000 of ExampleEmbodiment 2. The first moving direction estimation unit 2100 estimatesthe first moving direction using the first surveillance image 12 (S202).The second moving direction estimation unit 2120 estimates the secondmoving direction using the second surveillance image 22 (S204). Theimaging direction estimation unit 2140 estimates the imaging directionof the moving camera 20, based on the first moving direction, the secondmoving direction, the position and pose of the fixed camera 10, and theposition of the moving camera 20 (S206).

<Details of First Moving Direction Estimation Unit 2100>

The first moving direction estimation unit 2100 estimates the firstmoving direction which is the moving direction of the crowd in the firstsurveillance image 12 (S202). Here, there are various ways in which thefirst moving direction estimation unit 2100 estimates the first movingdirection. Hereinafter, the estimation method of the first movingdirection will be described.

«Method 1«

The first moving direction estimation unit 2100 calculates the opticalflow of pixels or feature points included in each of a plurality offirst surveillance images 12 arranged in time series. FIG. 20 is adiagram illustrating an optical flow calculated for the firstsurveillance image 12. Each arrow shown in FIG. 20 represents theoptical flow calculated for the first surveillance image 12.

The first moving direction estimation unit 2100 estimates the firstmoving direction, based on the calculated optical flow. For example, thefirst moving direction estimation unit 2100 selects one from the opticalflows, and handles the selected optical flow as the first movingdirection. For example, the first moving direction estimation unit 2100randomly selects one optical flow.

For example, the first moving direction estimation unit 2100 calculatesone vector by statistically processing the plurality of calculatedoptical flows, and handles the vector as the first moving direction. Thestatistical process is, for example, a process of calculating an averageof vectors.

Note that, since the technique for calculating the optical flow usingthe pixels or feature points included in the image is a well-knowntechnique, a detailed description of this technique will be omitted.

«Method 2«

The first moving direction estimation unit 2100 detects an objectcaptured in common in the plurality of first surveillance images 12arranged in time series, and estimates a first moving direction based ona change in the position of the object. FIG. 21 is a diagramillustrating a change in the position of an object. In FIG. 21 , theobject represented by the dotted line is captured in the t-th firstsurveillance image 12, and the object represented by the solid line iscaptured in the (t+1)-th first surveillance image 12. Arrows representchanges in the position of each object. The change in the position ofthe object is, for example, a vector connecting the centers of gravityof a plurality of regions representing the same object.

Note that, in a case where a plurality of objects are captured in thefirst surveillance image 12, as in the case of using the above-describedoptical flow, a plurality of vectors representing the change in theposition of the object are calculated. Therefore, for example, the firstmoving direction estimation unit 2100 selects one from a plurality ofobjects, and handles a vector representing a change in the position ofthe selected object as a first moving direction. For example, the firstmoving direction estimation unit 2100 randomly selects one object. Forexample, the first moving direction estimation unit 2100 selects thelargest object.

For example, the first moving direction estimation unit 2100 calculatesone vector by statistically processing a plurality of vectorsrepresenting the change in the position of a plurality of objects, andhandles the vector as the first moving direction. The statisticalprocess is, for example, a process of calculating an average of vectors.

«Method 3»

For example, the first moving direction estimation unit 2100 may detectthe moving direction of the crowd, based on the orientation of theobject captured in the first surveillance image 12. For example, in acase where the object is a person or an animal, the first movingdirection estimation unit 2100 determines the orientation of these facesand bodies, and handles the direction in which the face or the front ofthe body faces as the first moving direction. In a case where the objectis a car, motorbike, or object such as a flying object, the first movingdirection estimation unit 2100 determines a forwarding direction of theobject from the shape of the object captured in the first surveillanceimage 12 and the positions of various parts (such as a bumper and asteering wheel), and handles the determined forwarding direction as thefirst moving direction.

<Details of Second Moving Direction Estimation Unit 2120>

A method in which the second moving direction estimation unit 2120estimates the moving direction (second moving direction) of the crowdcaptured in the second surveillance image 22 is the same as the methodin which the first moving direction estimation unit 2100 estimates thefirst moving direction.

<About Reduction of Blurring>

In the first surveillance image 12 and the second surveillance image 22,the crowd may be blurred. For example, if the fixed camera 10 or themoving camera 20 images a moving crowd, the crowd may be blurred in eachsurveillance image. For example, in a case of performing imaging withthe fixed camera 10 while changing the pose of the fixed camera 10, orin a case of performing imaging with the moving camera 20 while changingthe pose or the position of the moving camera 20, the crowd may beblurred in each surveillance image.

Thus, it is preferable that the first moving direction estimation unit2100 estimates the first moving direction after applying a process forreducing blur (so-called image stabilization) to each first surveillanceimage 12. Likewise, it is preferable that the second moving directionestimation unit 2120 estimates the second moving direction afterapplying a process for reducing blur to each second surveillance image22.

<Details of Imaging Direction Estimation Unit 2140>

The imaging direction estimation unit 2140 estimates the imagingdirection of the moving camera 20, based on the first moving direction,the second moving direction, the position and pose of the fixed camera10, and the position of the moving camera 20 (S206). FIG. 22 is adiagram for explaining an operation of the imaging direction estimationunit 2140. Hereinafter, using FIG. 22 , a specific process in which theimaging direction estimation unit 2140 estimates the imaging directionof the moving camera 20 will be described.

First, the imaging direction estimation unit 2140 maps the first movingdirection on a plane (for example, on the map) of a plan view of theplace to be surveilled viewed in the vertical direction. In FIG. 22 ,the plane 70 is a plane of a plan view of the place to be surveilledviewed in the vertical direction. The moving direction 80 represents thedirection in which the moving direction of the crowd in the firstsurveillance image 12 is mapped to the plane 70. Note that, the plane 70is a plane having the upward direction as the north.

The imaging direction estimation unit 2140 calculates a moving direction(hereinafter, referred to as a third moving direction) on the plane 70of the object captured in the moving camera 20, using the movingdirection 80. In a case where the imaging range of the fixed camera 10and the imaging range of the moving camera 20 overlap, the third movingdirection is the same as the moving direction 80. FIG. 22 shows a casewhere the imaging range of the fixed camera 10 and the imaging range ofthe moving camera 20 overlap. A process for a case where the imagingrange of the fixed camera 10 and the imaging range of the moving camera20 do not overlap will be described later.

The imaging direction estimation unit 2140 determines an imagingdirection of the moving camera 20 as the estimation result, from aplurality of candidates (hereinafter, referred to as candidate imagingdirections) of the imaging direction of the moving camera 20. First, theimaging direction estimation unit 2140 calculates the moving direction(hereinafter, referred to as a candidate moving direction) of the crowdcaptured in the moving camera 20, which is assumed to be observed whenviewing the candidate imaging direction from the position of the movingcamera 20, for each candidate imaging direction.

In FIG. 22 , the candidate imaging direction 92 is eight directions:north, northeast, east, southeast, south, southwest, west, andnorthwest. The candidate moving directions 94-1 to 3 are candidatemoving directions in the case of viewing the candidate imagingdirections 92-1, 92-2, and 92-3 from the position of the moving camera20, respectively. Note that, in this example, since little or no crowdcan be viewed by viewing the candidate imaging directions other than theabove three directions from the position of the moving camera 20, thecandidate imaging directions other than the above three directions areomitted.

The imaging direction estimation unit 2140 performs matching betweeneach candidate moving direction and the second moving direction. In FIG.22 , the second moving direction is the second moving direction 95.Specifically, the imaging direction estimation unit 2140 estimates thatthe candidate imaging direction 92 corresponding to the candidate movingdirection 94 having a highest degree of equivalence to the second movingdirection 95 is the imaging direction of the moving camera 20. In FIG.22 , the candidate moving direction 94 having a highest degree ofequivalence to the second moving direction 95 is the candidate movingdirection 94-2. Therefore, the imaging direction estimation unit 2140estimates that the candidate imaging direction 92-2 corresponding to thecandidate moving direction 94-2 is the imaging direction of the movingcamera 20.

Note that, candidates of the imaging directions are not limited to theabove eight directions. For example, candidates of the imagingdirections may be four directions: north, east, south, and west.Further, the candidate imaging directions are not limited to thedirection for which generic names such as east, west, north and southare defined. For example, the imaging direction estimation unit 2140determines the candidate imaging direction with reference to thedirection of the sidewalk or the like in the vicinity of the movingcamera 20. FIG. 23 is a diagram for explaining a method of determining acandidate imaging direction with reference to a direction of a sidewalk.In FIG. 23 , the moving camera 20 is located near the sidewalk 110.Therefore, a surveillant or the like who surveils a crowd moving throughthe sidewalk 110 is considered to surveil the crowd, with the direction120, which is the normal direction of the sidewalk 110, as the frontdirection.

Therefore, for example, the imaging direction estimation unit 2140determines each candidate imaging direction with reference to thedirection 120, which is the normal direction of the direction of thesidewalk 110. For example, the imaging direction estimation unit 2140handles four directions of a direction 120, a direction 121 rotated fromthe direction 120 by +90 degrees, a direction 122 rotated from thedirection 120 by +180 degrees, and a direction 123 rotated from thedirection 120 by +270 degrees, as candidate imaging directions.

Note that, in a case of determining the candidate imaging directionbased on the sidewalk or the like which is a surveillance target, theimaging direction estimation unit 2140 has the above-mentioned mapinformation acquisition unit 2080, and uses the map information acquiredby the map information acquisition unit 2080.

«Regarding Case where Imaging Ranges of Fixed Camera 10 and MovingCamera 20 do not Overlap»

As described above, the imaging direction estimation unit 2140calculates a moving direction (a third moving direction) on the plane 70of the object captured in the moving camera 20, using the movingdirection 80. In the aforementioned example, it is assumed that theimaging range of the fixed camera 10 and the imaging range of the movingcamera 20 overlap. A case where the imaging range of the fixed camera 10and the imaging range of the moving camera 20 do not overlap will bedescribed below.

In a case where the imaging ranges of the fixed camera 10 and the movingcamera 20 do not overlap each other, the moving direction on the plane70 of the crowd captured in the first surveillance image 12 and themoving direction on the plane 70 of the crowd captured in the secondsurveillance image 22 are not always the same. FIG. 24 is a diagramillustrating a case where the imaging ranges of the fixed camera 10 andthe moving camera 20 do not overlap. In FIG. 24 , the direction in whichthe first moving direction is mapped onto the plane 70 is the movingdirection 80-1. On the other hand, the moving direction on the plane 70of the crowd captured in the moving camera 20 is not limited to themoving direction 80-2 which is the same direction as the movingdirection 80-1, but is also considered to be the moving direction 80-3or the like.

Thus, for example, the imaging direction estimation unit 2140 acquiresthe map information of the place to be surveilled, and calculates themoving direction of the crowd on the plane 70 captured by the movingcamera 20, based on the map information. Specifically, the imagingdirection estimation unit 2140 handles, as the moving direction on theplane 70 of the crowd captured by the moving camera 20, a movingdirection which is obtained by moving the moving direction on the plane70 of the crowd captured in the first surveillance image 12 to thevicinity of the moving camera 20 along a route (for example, a sidewalkor the like) along which the crowd is supposed to move on the map.

FIG. 25 is a diagram for explaining a method of estimating a movingdirection on the plane 70 of a crowd using map information. In thisexample, the object is a person. In FIG. 25 , the sidewalk 110 is asidewalk on which a person walks. In this case, when moving the movingdirection 80-1 (the moving direction on the plane 70 of the crowdcaptured in the first surveillance image 12) along the sidewalk 110, themoving direction on the plane 70 of the crowd in the vicinity of themoving camera 20 becomes the moving direction 80-3. Therefore, theimaging direction estimation unit 2140 handles the moving direction 80-3as the third moving direction. More specifically, the imaging directionestimation unit 2140 handles a vector that is the moving direction 80-1being moved along the line to the point where the distance between theline passing through the center of the sidewalk and the position of themoving camera 20 is shortest, as a third moving direction.

However, in a case where there is a plurality of moving routes of thecrowd that can be recognized from the map information, it may not beknown which moving route a crowd moves, in some cases. FIG. 26 is adiagram illustrating a case where the crowd has a plurality of movingroutes. In FIG. 26 , since the sidewalk 110 diverges, the crowd can movein either the moving direction 80-2 or the moving direction 80-3.

In such a case, the imaging direction estimation unit 2140 acquires themoving route information on the moving route of the crowd. To do so, thesurveillance information generation apparatus 2000 includes a movingroute information acquisition unit 2160. FIG. 27 is a block diagramillustrating the surveillance information generation apparatus 2000including the moving route information acquisition unit 2160.

The moving route information indicates a direction in which the crowdmoves. For example, in a case where the event is held at the eventvenue, it is considered that the crowd moves from the nearest station ofthe event venue to the event venue. In addition, in a case where eventsrequiring surveillance or the like are performed as described above, inorder to prevent an accident, crowd guidance is generally performed sothat the crowd moves to the event venue along a predetermined route.Thus, the moving route information indicates a predetermined movingroute of the crowd in this way.

FIG. 28 is a diagram exemplifying a process of estimating the movingdirection of the crowd using the moving route information. The movingroute information acquired in FIG. 28 is a map and informationindicating the moving route on the map. Even in the map indicated by themoving route information, the sidewalk 110 diverges, as in FIG. 26 .Here, a route 111 is described in the moving route information acquiredby the imaging direction estimation unit 2140. Thus, the imagingdirection estimation unit 2140 handles the moving direction 80-2 inwhich the moving direction 80-1 is moved along the route 111, as thethird moving direction.

Note that, there may be more than one piece of moving route informationabout a specific place. For example, in a case of the above-mentionedevent venue, the crowd moves from the nearest station to the event venuebefore the event starts, and the crowd moves from the event venue to thenearest station after the event ends. Thus, for example, the movingroute information acquisition unit 2160 acquires the combination of“moving route of the crowd and a time slot during which the movementoccurs” as moving route information. Here, it is assumed that the movingroute information is stored in advance in the storage apparatus or thelike accessible from the moving route information acquisition unit 2160.

Note that, in a case of using the moving route information, the imagingdirection estimation unit 2140 may not use the first surveillance image12 for estimation of the imaging direction. In this case, the imagingdirection estimation unit 2140 estimates the moving direction on theplane 70 of the crowd captured in the second surveillance image 22, fromthe moving route of the crowd indicated by the moving route information.For example, in FIG. 28 , the imaging direction estimation unit 2140 mayestimate that the moving direction on the plane 70 of the crowd capturedin the second surveillance image 22 is a moving direction 80-2, based onthe route 111, without using the moving direction 80-1.

«Narrowing Down of Candidate Imaging Direction»

Further, the imaging direction estimation unit 2140 may narrow down thecandidate imaging direction, before calculating the candidate movingdirection corresponding to the candidate imaging direction andperforming matching with the second moving direction. Thereby, since thenumber of times of matching can be reduced, there is an effect that itis possible to reduce the calculation amount of the process of theimaging direction estimation unit 2140. Hereinafter, several methods fornarrowing down candidate imaging directions will be exemplified.

<<<Narrowing Down Method Using Electronic Compass>>>

The imaging direction estimation unit 2140 narrows down candidateimaging directions using an electronic compass. Specifically, theimaging direction estimation unit 2140 excludes from the candidateimaging direction, a direction having a large difference in angle fromthe direction indicated by the electronic compass.

FIG. 29 is a diagram for explaining a method of narrowing down candidateimaging directions using an electronic compass. In FIG. 29 , thecandidate imaging direction is eight directions such as north andnorthwest. Further, the electronic compass shows northwest. In thiscase, due to lack of accuracy of the electronic compass, it is likelythat the imaging direction of the actual moving camera 20 is north orwest. On the other hand, it is unlikely that the south east, which isthe opposite of the direction indicated by the electronic compass, isthe imaging direction of the moving camera 20.

Thus, the imaging direction estimation unit 2140 excludes, from thecandidate imaging direction, the southeast or the like having a largedifference in angle from the direction indicated by the electroniccompass. Here, it is assumed that a predetermined number representingthe number of directions excluded from the candidate imaging directionis determined in advance. If the predetermined number is, for example,3, the imaging direction estimation unit 2140 excludes southeast, south,and east from the candidate imaging direction. If the predeterminednumber is 5, the imaging direction estimation unit 2140 excludessoutheast, south, east, southwest, and northeast from the candidateimaging direction. In FIG. 29 , southeast, south, and east are excludedfrom the candidate imaging direction.

<<<Narrowing Down Method Using Background Captured in SecondSurveillance Image 22>>>

The imaging direction estimation unit 2140 narrows down the candidateimaging direction, based on the background captured in the secondsurveillance image 22. For example, the imaging direction estimationunit 2140 excludes candidate imaging directions in which the backgroundcaptured in the second surveillance image 22 is predicted not to beincluded in the angle of view of the moving camera 20.

FIG. 30 is a diagram for explaining a method of narrowing down candidateimaging directions based on the background captured in the secondsurveillance image 22. FIG. 30 shows a map of the surroundings of asurveillance target. Here, it is assumed that the building 160 iscaptured in the second surveillance image 22. In this case, for example,in a case where the imaging direction of the moving camera 20 is in thesoutheast, the building 160 is not captured in the second surveillanceimage 22. On the other hand, if the imaging direction of the movingcamera 20 is the northwest, it is considered that the building 160 iscaptured in the second surveillance image 22.

Thus, the imaging direction estimation unit 2140 extracts backgroundsrepresenting characteristic buildings, signboards, and the like aroundthe surveillance target, from the background captured in the secondsurveillance image 22 in this way. Then, the imaging directionestimation unit 2140 excludes candidate imaging directions with a lowpossibility of capturing the extracted background, from the relationshipbetween the position of the extracted background on the map and theposition of the moving camera 20. Specifically, the imaging directionestimation unit 2140 excludes a candidate imaging direction that islargely different in angle from the direction starting from the positionof the moving camera 20 and ending at the position on the map of theextracted background. For example, the imaging direction estimation unit2140 excludes a predetermined number of candidate imaging directions,similar to the excluding of the candidate imaging direction using theelectronic compass described above. In FIG. 30 , southeast, south, andeast are excluded from the candidate imaging direction.

For example, the imaging direction estimation unit 2140 may also narrowdown the candidate imaging direction, based on the position of theextracted background in the second surveillance image 22. FIGS. 31A and31B are diagrams for explaining a method of narrowing down candidateimaging directions based on the position of a specific background on thesecond surveillance image 22. FIG. 31A shows the second surveillanceimage 22, and FIG. 31B shows the positional relationship between themoving camera 20 and the building 160 on the map. In the secondsurveillance image 22 of FIG. 31A, the building 160 is captured on theright side of the center of the second surveillance image 22. In thiscase, the imaging direction of the moving camera 20 is a directionrotated by a + angle from the direction (direction 170) connecting theposition of the moving camera 20 and the position of the building 160 onthe map.

Thus, the imaging direction estimation unit 2140 excludes candidateimaging directions in which the angle formed by the direction connectingthe position of the moving camera 20 and the position of the building160 on the map is in the range between −0 degrees and −180 degreesincluding −0 degrees and −180 degrees. For example, in the case of FIG.31B, the candidate imaging direction 174 is excluded from the candidateimaging directions, and the candidate imaging direction 172 is notexcluded from the candidate imaging directions.

«Comprehensive Estimation of Imaging Direction of Moving Camera 20»

The imaging direction estimation unit 2140 may estimate the imagingdirection of the moving camera 20 a plurality of times within apredetermined period, and comprehensively estimate the imaging directionof the moving camera 20, based on the plurality of estimation results.For example, it is assumed that the imaging direction estimation unit2140 calculates the comprehensive estimation result of the imagingdirection of the moving camera 20 every second. Further, it is assumedthat the fixed camera 10 and the moving camera 20 are cameras performingimaging at a frequency of 30 frames per second (30 fps). In this case,the imaging direction estimation unit 2140 estimates the imagingdirection of the moving camera 20 every time when the first surveillanceimage 12 and the second surveillance image 22 are generated, therebyestimating the imaging direction of the moving camera 20 30 times persecond. A comprehensive estimation result is calculated, based on theestimation result of 30 times.

Specifically, the imaging direction estimation unit 2140 statisticallyprocesses a plurality of estimation results calculated during apredetermined period to estimate the imaging direction of the movingcamera 20. For example, the imaging direction estimation unit 2140handles the mode of the plurality of estimation results as the imagingdirection of the moving camera 20. As an example, it is assumed that thebreakdown of the estimation results made 30 times per second is “north:20 times, northwest: 8 times, northeast: 2 times”. In this case, theimaging direction estimation unit 2140 estimates that the mostfrequently calculated north is the imaging direction of the movingcamera 20 during one second.

For example, the imaging direction estimation unit 2140 may calculate anaverage value of a plurality of estimation results and may handle theaverage value as the imaging direction of the moving camera 20.Specifically, the imaging direction estimation unit 2140 represents eachof the estimated directions of the moving camera 20 calculated aplurality of times by a numerical value with east as +0 degrees,calculates the average value of these numerical values, and handlesthese numerical values as the imaging direction of the moving camera 20.

Note that, in the case of calculating a comprehensive estimation resultby using estimation results performed a plurality of times during apredetermined period, the imaging direction estimation unit 2140 maycalculate the comprehensive estimation result by using only the resultswith high reliability among a plurality of estimation results. Forexample, it is assumed that the imaging direction estimation unit 2140calculates the comprehensive estimation result of the imaging directionof the moving camera 20 once per second. It is assumed that the fixedcamera 10 and the moving camera 20 are cameras performing imaging at 30fps.

In this case, the imaging direction estimation unit 2140 estimates theimaging direction of the moving camera 20 every time when the firstsurveillance image 12 and the second surveillance image 22 aregenerated, thereby estimating the imaging direction of the moving camera20 30 times per second. Next, the imaging direction estimation unit 2140divides the estimation results of 30 times per second into groups of tenconsecutive times in time series. Then, the imaging direction estimationunit 2140 calculates the comprehensive estimation result of the imagingdirection of the moving camera 20 during one second, using a group withhigh reliability among three groups.

The above-mentioned reliability of a group is determined, for example,based on the magnitude of variance of the estimation result within thegroup. In a case where the variance of the estimation result is large,since the estimated imaging direction of the moving camera 20 varieslargely, the reliability of the estimation result is considered to below. On the other hand, in a case where the variance of the estimationresult is small, since the estimated imaging direction of the movingcamera 20 varies small, the reliability of the estimation result isconsidered to be high. Therefore, for example, the imaging directionestimation unit 2140 calculates variance of the estimation result withinthe group by representing each estimation result as a numerical valuewith east as +0 degrees. Then, the imaging direction estimation unit2140 calculates the comprehensive estimation result, by using only theestimation result included in the group in which the calculated varianceis the predetermined value or less.

<<<Use of Change in Flow of Crowd>>>

Depending on the place, the flow of the crowd may change periodically orirregularly.

For example, in the vicinity of the intersection, the flow of the crowdchanges according to switching of traffic lights. FIGS. 32A and 32B arediagrams illustrating an example in which the flow of the crowd changesnear an intersection. In FIGS. 32A and 32B, a scene in which theintersection in plan view in the vertical direction is shown. At acertain time, it is assumed that the signal 131 of a pedestrian crossing130 is blue and the signal 141 of a pedestrian crossing 140 is red. Inthis case, the crowd flows to, for example, the direction 132 because itcrosses the pedestrian crossing 130. Thereafter, it is assumed that thesignal 131 becomes red and the signal 141 becomes blue. In this case,the crowd flows to, for example, the direction 142 because it crossesthe pedestrian crossing 140.

In a case where the flow of the crowd changes in this way, the imagingdirection estimation unit 2140 may estimate the imaging direction of themoving camera 20 for each of before and after the change, and estimatethe imaging direction of the moving camera 20 comprehensively from theresult. Specifically, the imaging direction estimation unit 2140estimates the imaging direction of the moving camera 20 for each ofbefore and after the crowd flow, by using the first surveillance image12 and the second surveillance image 22 imaged before and after thechange in the crowd flow. Then, the imaging direction estimation unit2140 calculates the final estimation result by statistically processingthe estimated imaging direction.

For example, in the example of FIGS. 32A and 32B, if the imagingdirection estimation unit 2140 estimates the imaging direction of themoving camera 20 in a case where the crowd flows in the direction 132,it is assumed that candidate moving directions having a high degree ofequivalence to the second moving direction are north and northwest.Next, if the imaging direction estimation unit 2140 estimates theimaging direction of the moving camera 20 in a case where the crowdflows in the direction 142, it is assumed that candidate movingdirections having a high degree of equivalence to the second movingdirection are north and northeast. In this case, the imaging directionestimation unit 2140 estimates that north that has a high degree ofequivalence (which is the mode) to the second moving direction in bothof the case where the crowd flows in the direction 132 and the casewhere the crowd flows in the direction 142, as the imaging direction ofthe moving camera 20.

«Following Change in Imaging Direction of Moving Camera 20»

The imaging direction of the moving camera 20 may change. Therefore, itis preferable that the imaging direction estimation unit 2140 repeatedlyestimates the imaging direction of the moving camera 20. For example,the imaging direction estimation unit 2140 repeatedly estimates theimaging direction of the moving camera 20 at a frequency such as onceper second or once per 10 seconds.

However, once the imaging direction of the moving camera 20 has beenestimated by the method described above, the imaging directionestimation unit 2140 can estimate subsequent imaging directions of themoving camera 20, based on subsequent changes in the imaging directionof the moving camera 20. Specifically, the imaging direction estimationunit 2140 calculates a change in the imaging direction of the movingcamera 20, and is able to estimate the imaging direction of the movingcamera 20 after the change, based on the calculated change and theimaging direction of the moving camera 20 estimated before the change.For example, the imaging direction estimation unit 2140 estimates thatthe imaging direction of the moving camera 20 is north at time t. Next,it is assumed that after t1 second, the imaging direction estimationunit 2140 calculates that the imaging direction of the moving camera 20has changed by +45 degrees. In this case, the imaging directionestimation unit 2140 estimates that the imaging direction of the movingcamera 20 at time t+t1 is northwest.

Therefore, the imaging direction estimation unit 2140 may realize a partof the processes of estimating the imaging direction of the movingcamera 20, which are repeatedly performed, by the process of estimatingthe imaging direction of the moving camera 20 based on the change in theimaging direction of the moving camera 20. Hereinafter, the process ofestimating the imaging direction of the moving camera 20 using the firstmoving direction and the second moving direction is referred to as thefirst estimation process, and the process of estimating the imagingdirection of the moving camera 20 by calculating the change in theimaging direction of the moving camera 20 is referred to as the secondestimation process.

FIG. 33 is a diagram illustrating breakdown of an estimation process ofthe imaging direction of the moving camera 20 executed by the imagingdirection estimation unit 2140 in time series. In the example of FIG. 33, the imaging direction estimation unit 2140 estimates the imagingdirection of the moving camera 20 once a second. Here, the imagingdirection estimation unit 2140 repeats the process “after performing thefirst estimation process once, the second estimation process isperformed nine times”. Therefore, the frequency at which the firstestimation process is performed is once in 10 seconds.

Note that, as described above, the imaging direction estimation unit2140 may estimate the imaging direction of the moving camera 20 aplurality of times, and totally estimate the imaging direction of themoving camera 20 based on the estimation results of the plurality oftimes, in some cases. In this case, in FIG. 33 , a process of performingcomprehensive estimation of the imaging direction of the moving camera20 is represented as a single first estimation process.

There are various methods of calculating the change in the imagingdirection of the moving camera 20. For example, the change in theimaging direction of the moving camera 20 can be calculated using anacceleration sensor attached to the moving camera 20. By using theacceleration sensor attached to the moving camera 20, it is possible tocalculate the relative change of the pose of the moving camera 20, fromthe change of the output of the acceleration sensor. Thus, for example,the imaging direction estimation unit 2140 calculates a change in theimaging direction of the moving camera 20 at the time t+t1, from thedifference between the output of the acceleration sensor when estimatingthe imaging direction of the moving camera 20 at the time t and theoutput of the acceleration sensor at the time t+t1.

For example, a change in the imaging direction of the moving camera 20can be calculated by tracking the change of the feature points capturedin the second surveillance image 22. FIG. 34 is a diagram illustrating achange in the position of feature points of the second surveillanceimage 22. Here, it is assumed that the feature point captured at theposition 150-1 at a certain time t moves to the position 150-2 at thetime t+t1. Here, the moving distance in the horizontal direction is x.First, since the feature point moves to the right, it can be known thatthe imaging direction of the moving camera 20 changes in the +direction. Furthermore, the magnitude of the change in the imagingdirection of the moving camera 20 can be known, based on the distance xin the horizontal direction in which the feature point has moved and theangle of view of the moving camera 20. Therefore, the imaging directionestimation unit 2140 can calculate the change of the imaging directionof the moving camera 20, based on the direction in which the imagingdirection of the moving camera 20 has changed and the magnitude of thechange.

The calculation amount in the second estimation process (a process ofestimating the imaging direction of the moving camera 20 by utilizingthe change in the imaging direction of the moving camera 20) is smallerthan the calculation amount in the first estimation process (a processof calculating the first moving direction and the second movingdirection and estimating the imaging direction of the moving camera 20).Therefore, in a case of calculating the imaging direction of the movingcamera 20 repeatedly, there is an effect of reducing the processing loadof the surveillance information generation apparatus 2000 by using thefirst estimation process and the second estimation process together.

<About Correction of Inclination of Surveillance Image>

It is preferable that the first moving direction estimation unit 2100,the second moving direction estimation unit 2120, and the imagingdirection estimation unit 2140 perform the above-described process aftercorrecting the inclinations in the vertical direction of the firstsurveillance image 12 and the second surveillance image 22. For example,the correction of the inclination in the vertical direction of eachimage can be performed based on the inclination of the line of abuilding or the like captured in the image. For example, in a case wherethe building is captured in the first surveillance image 12, the firstmoving direction estimation unit 2100 or the like extracts the line inthe height direction of the building captured in the first surveillanceimage 12, and corrects the line so as to be perpendicular to thehorizontal direction of the first surveillance image 12 to correct theinclination in the vertical direction of the first surveillance image12.

In addition, correction of the inclination in the vertical direction ofthe first surveillance image 12 may be performed using the cameraparameter indicating the inclination in the vertical direction of thefixed camera 10. The correction of the inclination of the secondsurveillance image 22 in the vertical direction may be performed byusing the inclination of the moving camera 20 in the vertical directionthat can be calculated from the acceleration sensor attached to themoving camera 20.

<Hardware Configuration of Surveillance Information Generation Apparatus2000>

The surveillance information generation apparatus 2000 of ExampleEmbodiment 2 is realized using the computer 1000 in the same way as inExample Embodiment 1 (see FIG. 4 ). In the present example embodiment,each program module stored in the storage 1080 described above furtherincludes a program for realizing each function described in the presentexample embodiment.

Modification Example

An apparatus for estimating the imaging direction of the moving camera20 may be provided separately from the surveillance informationgeneration apparatus 2000. This apparatus is referred to as an imagingdirection estimation apparatus 3000. FIG. 35 is a block diagramillustrating the imaging direction estimation apparatus 3000. In FIG. 35, each block represents a functional unit configuration, instead of ahardware unit configuration.

The imaging direction estimation apparatus 3000 includes a firstsurveillance image acquisition unit 2020, a second surveillance imageacquisition unit 2040, a first moving direction estimation unit 2100, asecond moving direction estimation unit 2120, and an imaging directionestimation unit 2140. The function of each functional component is asdescribed above.

The hardware configuration of the imaging direction estimation apparatus3000 is illustrated, for example, in FIG. 4 , similar to thesurveillance information generation apparatus 2000. Note that, thestorage of the computer that implements the imaging direction estimationapparatus 3000 stores program modules for realizing the functions of therespective functional components shown in FIG. 35 .

Note that, in a case where an imaging direction estimation apparatus3000 is provided independently from the surveillance informationgeneration apparatus 2000, the generation unit 2060 of the surveillanceinformation generation apparatus 2000 acquires the imaging direction ofthe moving camera 20 from the imaging direction estimation apparatus3000 and uses it.

Although the example embodiments of the present invention have beendescribed above with reference to the drawings, these are examples ofthe present invention, and it is possible to use various configurationsother than the above exemplary embodiments.

Examples of a reference aspect will be added below.

1. A surveillance information generation apparatus comprising:

a first acquisition unit acquiring a first surveillance image imaged bya fixed camera, which is a camera a position of which is fixed;

a second acquisition unit acquiring a second surveillance image imagedby a moving camera, which is a camera a position of which is not fixed;and

a generation unit generating surveillance information of an object byusing the first surveillance image and the second surveillance image.

2. The surveillance information generation apparatus according to 1.,wherein the generation unit generates a display in which the secondsurveillance image is superimposed on the first surveillance image, asthe surveillance information.

3. The surveillance information generation apparatus according to 2.,wherein the generation unit superimposes the second surveillance imageon a position on the first surveillance image corresponding to aposition of the moving camera in a real world.

4. The surveillance information generation apparatus according to 2. or3., wherein the generation unit performs:

displaying a mark representing the moving camera on the firstsurveillance image; and

in a case where an operation to select the mark is performed, as thesurveillance information, generating a display in which the secondsurveillance image generated by the moving camera corresponding to themark is superimposed on the first surveillance image.

5. The surveillance information generation apparatus according to 1.,wherein the generation unit performs:

generating distribution information indicating distribution of an objectcaptured in the first surveillance image;

calculating the number of objects captured in the second surveillanceimage; and

correcting the distribution information by using the number of objectscaptured in the second surveillance image, and generating the correcteddistribution information as the surveillance information.

6. The surveillance information generation apparatus according to 5.,wherein the generation unit generates a display in which the correcteddistribution information is superimposed on the first surveillanceimage.

7. The surveillance information generation apparatus according to 5.,further comprising a map information acquisition unit acquiring mapinformation of a place to be surveilled,

wherein the generation unit generates a display in which the correcteddistribution information is superimposed on a map indicated by the mapinformation.

8. The surveillance information generation apparatus according to one of5. to 7., wherein the generation unit performs:

calculating an imaging range of the moving camera in the firstsurveillance image, with using an imaging direction of the movingcamera; and

correcting the number or distribution of objects within the imagingrange of the moving camera indicated by the distribution information,with using the number of objects captured in the second surveillanceimage.

9. The surveillance information generation apparatus according to anyone of 1. to 8., wherein the generation unit superimposes a displayrepresenting an imaging direction of the moving camera on the firstsurveillance image.

10. The surveillance information generation apparatus according to 8. or9., further comprising:

a first moving direction estimation unit estimating a first movingdirection, which is a moving direction of the object in the firstsurveillance image;

a second moving direction estimation unit estimating a second movingdirection, which is a moving direction of the object in the secondsurveillance image; and

an imaging direction estimation unit estimating the imaging direction ofthe moving camera, based on the first moving direction, the secondmoving direction, a position and a pose of the fixed camera, and aposition of the moving camera,

wherein the generation unit uses the estimated imaging direction of themoving camera as the imaging direction of the moving camera.

11. The surveillance information generation apparatus according to 10.,wherein the imaging direction estimation unit performs:

calculating a third moving direction which is the moving direction ofthe object captured in the second surveillance image, on a plane of aplan view of a place to be surveilled in a vertical direction, withusing the position and pose of the fixed camera and the first movingdirection;

calculating a plurality of candidate moving directions, which are movingdirections of the object moving in the third moving direction that areassumed to be observed when viewing each of the plurality of candidateimaging directions from the position of the moving camera; and

estimating that the candidate moving direction with a highest degree ofequivalence to the second moving direction is the imaging direction ofthe moving camera.

12. The surveillance information generation apparatus according to 11.,wherein the imaging direction estimation unit calculates the movingdirection of the object captured in the first surveillance image on theplane, using the position and pose of the fixed camera and the firstmoving direction, and handles the calculated moving direction as thethird moving direction.

13. The surveillance information generation apparatus according to 11.,further comprising a moving route information acquisition unit acquiringmoving route information indicating a moving route of an object,

wherein the imaging direction estimation unit calculates the movingdirection of the object captured in the first surveillance image on theplane, with using the position and pose of the fixed camera and thefirst moving direction, and

wherein a moving direction that is obtained by moving the calculatedmoving direction to the vicinity of the moving camera along the movingroute indicated by the moving route information is handled as the thirdmoving direction.

14. An imaging direction estimation apparatus comprising:

a first moving direction estimation unit estimating a first movingdirection, which is a moving direction of an object in a firstsurveillance image imaged by a fixed camera which is a camera whoseposition is fixed;

a second moving direction estimation unit estimating a second movingdirection, which is a moving direction of an object in a secondsurveillance image imaged by a moving camera which is a camera whoseposition is not fixed; and

an imaging direction estimation unit estimating the imaging direction ofthe moving camera, based on the first moving direction, the secondmoving direction, a position and a pose of the fixed camera, and aposition of the moving camera.

15. The imaging direction estimation apparatus according to 14.,

wherein the imaging direction estimation unit performs:

calculating a third moving direction which is the moving direction ofthe object captured in the second surveillance image, on a plane of aplan view of a place to be surveilled in a vertical direction, withusing the position and pose of the fixed camera and the first movingdirection;

calculating a plurality of candidate moving directions which are movingdirections of the object when viewing the object moving from each of theplurality of candidate imaging directions to the third moving directionat the position of the moving camera; and

estimating that the candidate moving direction with a highest degree ofequivalence to the second moving direction is the imaging direction ofthe moving camera.

16. The imaging direction estimation apparatus according to 15., whereinthe imaging direction estimation unit calculates the moving direction ofthe object captured in the first surveillance image on the plane, usingthe position and pose of the fixed camera and the first movingdirection, and handles the calculated moving direction as the thirdmoving direction.

17. The imaging direction estimation apparatus according to 15., furthercomprising a moving route information acquisition unit acquiring movingroute information indicating a moving route of an object,

wherein the imaging direction estimation unit calculates the movingdirection of the object captured in the first surveillance image on theplane, using the position and pose of the fixed camera and the firstmoving direction, and

wherein a moving direction that is obtained by moving the calculatedmoving direction to the vicinity of the moving camera along the movingroute indicated by the moving route information is handled as the thirdmoving direction.

18. A surveillance information generation method executed by a computer,the method comprising:

a first acquisition step of acquiring a first surveillance image imagedby a fixed camera, which is a camera a position of which is fixed;

a second acquisition step of acquiring a second surveillance imageimaged by a moving camera, which is a camera a position of which is notfixed; and

a generation step of generating surveillance information of an object byusing the first surveillance image and the second surveillance image.

19. The surveillance information generation method according to 18.,wherein in the generation step, a display in which the secondsurveillance image is superimposed on the first surveillance image isgenerated as the surveillance information

20. The surveillance information generation method according to 19.,wherein in the generation step, the second surveillance image issuperimposed on a position on the first surveillance image correspondingto a position of the moving camera in a real world.

21. The surveillance information generation method according to 19. or20., wherein in the generation step,

displaying a mark representing the moving camera on the firstsurveillance image, and

in a case where an operation to select the mark is performed, as thesurveillance information, generating a display in which the secondsurveillance image generated by the moving camera corresponding to themark is superimposed on the first surveillance image.

22. The surveillance information generation method according to 18.,wherein in the generation step,

generating distribution information indicating distribution of an objectcaptured in the first surveillance image,

calculating the number of objects captured in the second surveillanceimage, and

correcting the distribution information by using the number of objectscaptured in the second surveillance image, and generating the correcteddistribution information as the surveillance information.

23. The surveillance information generation method according to 22.,wherein in the generation step, generating a display in which thecorrected distribution information is superimposed on the firstsurveillance image.

24. The surveillance information generation method according to 22.,further comprising a map information acquisition step of acquiring mapinformation of a place to be surveilled,

wherein in the generation step, generating a display in which thecorrected distribution information is superimposed on a map indicated bythe map information.

25. The surveillance information generation method according to any oneof 22. to 24., wherein in the generation step,

calculating an imaging range of the moving camera in the firstsurveillance image, using an imaging direction of the moving camera, and

correcting the number or distribution of objects within the imagingrange of the moving camera indicated by the distribution information,with using the number of objects captured in the second surveillanceimage.

26. The surveillance information generation method according to any oneof 18. to 25., wherein in the generation step, superimposing a displayrepresenting an imaging direction of the moving camera on the firstsurveillance image.

27. The surveillance information generation method according to 25. or26., further comprising:

a first moving direction estimation step of estimating a first movingdirection, which is a moving direction of the object in the firstsurveillance image;

a second moving direction estimation step of estimating a second movingdirection, which is a moving direction of the object in the secondsurveillance image; and

an imaging direction estimation step of estimating the imaging directionof the moving camera, based on the first moving direction, the secondmoving direction, a position and a pose of the fixed camera, and aposition of the moving camera,

wherein in the generation step, the estimated imaging direction of themoving camera is used as the imaging direction of the moving camera.

28. The surveillance information generation method according to 27.,wherein in the imaging direction estimation step,

calculating a third moving direction which is the moving direction ofthe object captured in the second surveillance image, on a plane of planview of a place to be surveilled in a vertical direction, with using theposition and pose of the fixed camera and the first moving direction,

calculating a plurality of candidate moving directions which are movingdirections of the object moving in the third direction that are assumedto be observed when viewing each of the plurality of candidate imagingdirections from the position of the moving camera, and

estimating the candidate moving direction with a highest degree ofequivalence to the second moving direction is estimated as the imagingdirection of the moving camera.

29. The surveillance information generation method according to 28.,wherein calculating in the imaging direction estimation step, the movingdirection of the object captured in the first surveillance image on theplane, with using the position and pose of the fixed camera and thefirst moving direction, and handling the calculated moving direction asthe third moving direction.

30. The surveillance information generation method according to 28.,further comprising:

moving route information acquisition step of acquiring moving routeinformation indicating a moving route of an object,

wherein in the imaging direction estimation step, calculating the movingdirection of the object captured in the first surveillance image on theplane with using the position and pose of the fixed camera and the firstmoving direction, and

wherein a moving direction that is obtained by moving the calculatedmoving direction to the vicinity of the moving camera along the movingroute indicated by the moving route information is handled as the thirdmoving direction.

31. An imaging direction estimation method executed by a computer, themethod comprising:

a first moving direction estimation step of estimating a first movingdirection which is a moving direction of an object in a firstsurveillance image imaged by a fixed camera, which is a camera aposition of which is fixed;

a second moving direction estimation step of estimating a second movingdirection which is a moving direction of an object in a secondsurveillance image imaged by a moving camera, which is a camera aposition of which is not fixed; and

an imaging direction estimation step of estimating the imaging directionof the moving camera, based on the first moving direction, the secondmoving direction, a position and a pose of the fixed camera, and aposition of the moving camera.

32. The imaging direction estimation method according to 31.,

wherein in the imaging direction estimation step,

calculating a third moving direction which is the moving direction ofthe object captured in the second surveillance image, on a plane of aplan view of a place to be surveilled in a vertical direction, withusing the position and the pose of the fixed camera and the first movingdirection,

calculating a plurality of candidate moving directions which are movingdirections of the object moving in the third moving direction that areassumed to be observed when viewing each of the plurality of candidateimaging directions from the position of the moving camera, and

estimating the candidate moving direction with a highest degree ofequivalence to the second moving direction as the imaging direction ofthe moving camera.

33. The imaging direction estimation method according to 32., wherein inthe imaging direction estimation step, calculating the moving directionof the object captured in the first surveillance image on the plane,with using the position and the pose of the fixed camera and the firstmoving direction, and handling the calculated moving direction as thethird moving direction.

34. The imaging direction estimation method according to 32., furthercomprising moving route information acquisition step of acquiring movingroute information indicating a moving route of an object,

wherein in the imaging direction estimation step,

calculating the moving direction of the object captured in the firstsurveillance image on the plane with using the position and pose of thefixed camera and the first moving direction, and

wherein a moving direction that is obtained by moving the calculatedmoving direction to the vicinity of the moving camera along the movingroute indicated by the moving route information is handled as the thirdmoving direction.

35. A program that causes a computer to execute each step according toany one of 18. to 34.

The invention claimed is:
 1. A surveillance system comprising: at leastone memory storing instructions; and at least one processor coupled tothe at least one memory, the at least one processor being configured toexecute the instructions to: acquire a first surveillance image capturedby a fixed camera which has a fixed position; acquire a secondsurveillance image captured by a wearable camera which does not have afixed position; acquire position information indicating a position ofthe wearable camera; and superimpose a first mark indicating the fixedcamera onto a displayed map at a location corresponding to a position ofthe fixed camera, a second mark indicating the wearable camera onto thedisplayed map at a location corresponding to a position of the wearablecamera, and the second surveillance image which is captured by thewearable camera onto the displayed map, the first mark being a differentmark from the second mark.
 2. The surveillance system according to claim1, wherein the at least one processor is further configured tosuperimpose the first surveillance image on the displayed map.
 3. Thesurveillance system according to claim 1, wherein the at least oneprocessor is further configured to calculate the location correspondingto the position of the wearable camera by using the position informationof the wearable camera and information of a place under surveillanceindicated by the map information.
 4. The surveillance system accordingto claim 1, wherein the at least one processor is further configured tospecify the location corresponding to the position of the wearablecamera by using the position information transmitted from the wearablecamera separately from the second surveillance image.
 5. Thesurveillance system according to claim 1, wherein the at least oneprocessor is further configured to: receive an input of a selection of amark from among a plurality of second marks indicating wearable cameras;and superimpose, onto the displayed map, the second surveillance imagegenerated by the wearable camera corresponding to the selected mark at alocation corresponding to a position of the wearable cameracorresponding to the selected mark.
 6. The surveillance system accordingto claim 1, wherein the at least one processor is further configured to:receive, after displaying the second surveillance image, an inputoperation for moving the second surveillance image; and change thelocation of the superimposed second surveillance image.
 7. Thesurveillance system according to claim 1, wherein the at least oneprocessor is further configured to: specify positions of wearablecameras in a real world based on the position information indicating aposition of each of the respective wearable cameras; superimpose, ontothe map, third marks with which signs distinguishable from each otherare accompanied, locations of the third marks corresponding to thespecified positions; display a plurality of second surveillance images;and display, on each of the plurality of second surveillance images, asign out of the signs accompanied with a third mark displayed at alocation corresponding to the wearable camera that captured the secondsurveillance image.
 8. A surveillance method comprising: acquiring afirst surveillance image captured by a fixed camera which has a fixedposition; acquiring a second surveillance image captured by a wearablecamera which does not have a fixed position; acquiring positioninformation indicating a position of the wearable camera; andsuperimposing a first mark indicating the fixed camera onto a displayedmap at a location corresponding to a position of the fixed camera, asecond mark indicating the wearable camera onto the displayed map at alocation corresponding to a position of the wearable camera, and thesecond surveillance image which is captured by the wearable camera ontothe displayed map, the first mark being a different mark from the secondmark.
 9. The surveillance method according to claim 8, furthercomprising: superimposing the first surveillance image on the displayedmap.
 10. The surveillance method according to claim 8, furthercomprising: calculating the location corresponding to the position ofthe wearable camera by using the position information of the wearablecamera and information of a place under surveillance indicated by themap information.
 11. The surveillance method according to claim 8,further comprising: specifying the location corresponding to theposition of the wearable camera by using the position informationtransmitted from the wearable camera separately from the secondsurveillance image.
 12. The surveillance method according to claim 8,further comprising: receiving an input of a selection a mark from amonga plurality of second marks indicating wearable cameras; andsuperimposing, onto the displayed map, the second surveillance imagegenerated by the wearable camera corresponding to the selected mark at alocation corresponding to a position of the wearable cameracorresponding to the selected mark.
 13. The surveillance methodaccording to claim 8, further comprising: receiving, after displayingthe second surveillance image, an input operation for moving the secondsurveillance image; and changing the location of the superimposed secondsurveillance image.
 14. The surveillance method according to claim 8,further comprising: specifying positions of wearable cameras in a realworld based on the position information indicating a position of each ofthe respective wearable cameras; superimposing, onto the map, thirdmarks with which signs distinguishable from each other are accompanied,locations of the third marks corresponding to the specified positions;displaying a plurality of second surveillance images; and displaying, oneach of the plurality of second surveillance images, a sign out of thesigns accompanied with a third mark displayed at a locationcorresponding to the wearable camera that captured the secondsurveillance image.
 15. A non-transitory computer-readable storagemedium storing a program that causes a computer to perform: acquiring afirst surveillance image captured by a fixed camera which has a fixedposition; acquiring a second surveillance image captured by a wearablecamera which does not have a fixed position; acquiring positioninformation indicating a position of the wearable camera; andsuperimposing a first mark indicating the fixed camera onto a displayedmap at a location corresponding to a position of the fixed camera, asecond mark indicating the wearable camera onto the displayed map at alocation corresponding to a position of the wearable camera, and thesecond surveillance image which is captured by the wearable camera ontothe displayed map, the first mark being a different mark from the secondmark.
 16. The storage medium according to claim 15, wherein the programfurther causes the computer to perform: superimposing the firstsurveillance image on the displayed map.
 17. The storage mediumaccording to claim 15, wherein the program further causes the computerto perform: calculating the location corresponding to the position ofthe wearable camera by using the position information of the wearablecamera and information of a place under surveillance indicated by themap information.
 18. The storage medium according to claim 15, whereinthe program further causes the computer to perform: specifying thelocation corresponding to the position of the wearable camera by usingthe position information transmitted from the wearable camera separatelyfrom the second surveillance image.
 19. The storage medium according toclaim 15, wherein the program further causes the computer to perform:receiving an input of a selection a mark from among a plurality ofsecond marks indicating wearable cameras; and superimposing, onto thedisplayed map, the second surveillance image generated by the wearablecamera corresponding to the selected mark at a location corresponding toa position of the wearable camera corresponding to the selected mark.20. The storage medium according to claim 15, wherein the programfurther causes the computer to perform: receiving, after displaying thesecond surveillance image, an input operation for moving the secondsurveillance image; and changing the location of the superimposed secondsurveillance image.